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On December 13 lh 1998, the Total Ozone Mapping Spectrometer - Earth 
Probe (TOMS-EP) spacecraft experienced a Single Event Upset which 
caused the system to reconfigure and enter a Safe Mode. This incident 
occurred two and a half years after the launch of the spacecraft which 
was designed for a two year life. A combination of factors, including 
changes in component behavior due to age and extended use, very 
unfortunate initial conditions and the safe mode processing logic 
prevented the spacecraft from entering its nominal long term storage 
mode. The spacecraft remained in a high fuel consumption mode 
designed for temporary use. By the time the onboard fuel was exhausted, 
the spacecraft was Sun pointing in a high rate flat spin. 

Although the uncontrolled spacecraft was initially in a power and thermal 
safe orientation, it would not stay in this state indefinitely due to a slow 
precession of its momentum vector. A recovery team was immediately 
assembled to determine if there was time to develop a method of de- 
spinning the vehicle and return it to normal science data collection. A 
three stage plan was developed that used the onboard magnetic torque 
rods as actuators. The first stage was designed to reduce the high spin 
rate to within the linear range of the gyros. The second stage transitioned 
the spacecraft from sun pointing to orbit reference pointing. The final 
stage returned the spacecraft to normal science operation. The entire 
recovery scenario was simulated with a wide range of initial conditions to 
establish the expected behavior. The recovery sequence was started on 
December 28 <h 1998 and completed by December 31 st . TOMS-EP was 
successfully returned to science operations by the beginning of 1999. 

This paper describes the TOMS-EP Safe Mode design and the factors 
which led to the spacecraft anomaly and loss of fuel. The recovery and 
simulation efforts are described. Flight data are presented which show 
the performance of the spacecraft during its return to science. Finally, 
lessons learned are presented. 


’ Prepared for technical papers that may later be published in the proceedings of the American Astronautical Society. 



INTRODUCTION 


The Total Ozone Mapping Spectrometer - Earth Probe (TOMS-EP) is a National 
Aeronautics and Space Administration (NASA) mission to continue the long-term daily 
mapping of the global distribution of Earth’s atmospheric ozone layer. The satellite was 
built by TRW for NASA's Goddard Space Flight Center. TOMS-EP collects high 
resolution measurements of the total column of ozone. The NASA-developed instrument 
measures ozone directly by mapping ultraviolet light emitted by the Sun to that scattered 
from the Earth’s atmosphere back to the satellite. The TOMS instrument has mapped in 
detail the global ozone distributions as well as the Antarctic “ozone hole,” which forms 
September through November of each year. In addition, TOMS measures sulfur-dioxide 
released in volcanic eruptions which may be used to detect volcanic ash clouds that are 
hazardous to commercial aviation. 

TOMS-EP was inserted into orbit by the Pegasus XL booster on July 2, 1996. In the nine 
days following launch, the spacecraft executed a series of Delta V bums to reach a 500 
km circular Sun-synchronous mission orbit with an ascending node mean local time 
crossing of 11:18 AM. Originally, the data obtained from TOMS-EP were intended to 
complement data obtained from ADEOS TOMS, which gave complete equatorial 
coverage due to its higher orbit. However, with the failure of ADEOS in June 1997, the 
orbit of TOMS-EP was boosted to 740 km and circularized to provide coverage that is 
almost daily. TOMS-EP is currently the only satellite providing scientific data with an 
operating TOMS instrument. A QuickTOMS mission is planned for launch in August, 
2000 with another TOMS instrument. Figure 1 illustrates the TOMS-EP satellite. 
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SYSTEM SAFE MODES 


To understand the anomaly, it is necessary to understand the system implementation of 
the active safe modes. The Safe Power Mode uses all standby redundant equipment. It 
has two submodes, Sun Point Recovery and Long Term Hold, whose functions are 
defined in Table 1. Both submodes point the +X (roll) spacecraft axis to the Sun. The 
coarse sun sensor assembly (CSSA) is used for pitch and yaw attitude error and a single 
two-axis gyro provides rate information about pitch and yaw. The spacecraft undergoes 
an open loop roll spin-up by two 1 pound hydrazine thrusters prior to entering Long Term 
Hold. 


Table 1 

Safe Power Submodes 


Mode 

Submode 

Description 

Automatic Transitions 

Safe Power 

Sun Point Recovery 

Two axis inertial sun pointing 
mode. CSSA and gyro are used 
as sensors. Thrusters used as 
actuators 

Entry from any other mode 
due to fault condition. Entry 
fromJ_ong Term Hold due to 
excessive Sun pointing error. 

Long Term Hold 

i 

Spin stabilized Sun pointing 
precession control mode with 
two axis rate control. CSSA and 
gyro are used as sensors. 
Thrusters used as actuators 

Entry from Sun Point 
Recovery only after successful 
Sun acquisition. Exit from 
mode if there is excessive Sun 
pointing error. 


ANOMALY OUTLINE 

The anomaly began when an event caused the spacecraft to transition from the prime 
processor to the redundant processor in response to a critical parameter that exceeded an 
established limit. The spacecraft successfully aligned the +X axis with the sun line using 
a two axis inertial controller based on processed coarse sun sensor measurements and a 
single two axis rate gyro. At this point, the flight software should automatically spin up 
the spacecraft about the roll axis and transition to a very low fuel consumption 
momentum based controller. At some point in the transition, the flight software failed to 
complete the transfer to the momentum based controller. Table 2 provides a concise 
timeline of events starting just before the processor reboot. 

Within approximately 6 hours from entering Safe Power Mode, TOMS-EP had used 
virtually all of the 25 lb of Hydrazine fuel that remained before the anomaly. The 
spacecraft was pointed at the Sun, but was uncontrolled and spinning at approximately 18 
deg/sec about the +X (roll) axis. 

The large amount of thruster activity had a small effect on the TOMS-EP orbit. TOMS- 
EP is required to stay within an ascending node crossing time of between 11:03 and 
11:30. Before the anomaly, ascending node crossing time evolution was not a science 
life-limiting factor. After the anomaly, the rate of change of the ascending node crossing 
time was increased by about 3.6 min per year. This rate of change still allows more than 
4 years of operation before the ascending node crossing time begins to degrade science 
collection. 
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Table 2 

Anomaly Timeline 


Event 

Time 

Notes 

Corrupted Ephemeris Position (ECI) Data 
In Telemetry. 

347/15:1 1:26 

Previous value of position was 2139.6, 4193.99, -5330.01. 
Position reading at this time was 1042.67, 5484.46, 4412.70. 

Large Pitch Error. 

347/15:11:30 

Error is calculated by subtracting the onboard propagated 
position quaternion from the commanded quaternion. Since error 
did not appear in either roll or yaw, suspect variables for SEU 
are those related to time (onboard clock, software time or epoch 
time). 

First Thruster Firing To Counter Wheel 
Spin Down. 

347/15: 13:06 

The first thruster activity occurs more than 1.5 minutes after the 
Redundant Processor boot is finished. This is the required time 
to configure the ADCS hardware and initialize Sun Point 
Recovery. Pitch thruster firings seem to be very clean. The 
system started virtually sun pointed. Correct thruster pair 
participate in the removal of wheel momentum as it bleeds into 
the spacecraft. 

End of minimum ten minute window 
required in Sun Point Recover. 

347/15:23:07 

The safing logic waits a minimum of 10 minutes in Sun Point 
Recovery to allow the wheels to run down. This should prevent 
momentum coupling while the spacecraft spins up. 

System begins to monitor the five 
required conditions necessary to begin the 
transition from two single axis inertial 
controllers to a spin stabilized momentum 
controller. 

347/15:23:07 

The five conditions required to start the transition are: 

1 . No presence in Fine Sun Sensor #2 

2. Pitch rate within specified threshold 

3. Yaw rate within specified threshold 

4. Pitch angle within specified threshold 

5. Yaw angle within specified threshold 

At this time, the processed telemetry showed that all five of the 
conditions above were satisfied. The flight software changes the 
flag “runup” from 0 (as initialized) to 1 to denote that the system 
is ready to be spun up. 

Start of Roll Spin-up 

347/15:23:07 

Immediately after the minimum time window, the roll thrusters 
begin to spin up the spacecraft. Telemetry from the thruster 
commands shows the total roll on time to be approximately 
19.15 seconds. The expected roll rate with this duration pulse 
should be 3.9 to 4.5 deg/sec. This matches with the algorithm in 
the flight software and the tank reading in telemetry of 36 counts 
(8 bit reading) which represents 85 psi. At the start of the roll 
spin up, the flight software sets the flag “runup” to 2 to let the 
system know that the roll spin-up has started. 

Completion of roll spin up / transition to 
spin stabilized controller. 

347/15:24:04 

The telemetry shows that the roll spin-up completed on time and 
yet the system failed transition to the spin stabilized controller. 

Continuous Firing of Pitch Thrusters. 

347/15:24:04 

Once there was angular velocity in the roll axis, imperfections in 
the alignment of the inertial and control axes caused a constant 
pitch rate to appear on the pitch gyro. The inertial control law 
continuously fired the pitch thrusters to compensate for this rate. 
The thrusters were ineffective due to the spinning dynamics. A 
small torque coupling between pitch and roll resulted in a 
continuous increase in roll rate as the pitch thrusters were fired. 

1 st Contact after anomaly. 

Ground acquires downlink with only 3 
min to Horizon LOS. 

347/16:01:00 

Ground observes spacecraft in Sun Point Recovery. 
Tank pressure 84 psi. 

First expiration of sun acquisition 
timeout. 

347/17:08:11 

The failure to reach the spin stabilized mode caused the 
Redundant Processor to reset after 7000 seconds and attempt to 
acquire the Sun again in Sun Point Recovery. This was the first 
of three or four resets due to this trigger. The subsequent 
attempts to acquire the Sun failed due to the system dynamics. 

2 nd Contact. 

347/17:44:00 

Ground observes Sun Point Recovery failure to acquire. 
Tank pressure 78 psi. 

3 rd Contact. 

347/19:19 1 

Ground evaluating problem. 

4 m Contact. 

347/20:57 

Ground turns on GRA 1 & 2. 

Spacecraft processor reset occurs during pass. Tank pressure 77 
Psi 

5 th Contact. 

347/22:38 

Tank pressure 9 psi. 

Spacecraft spinning at 18 deg/sec. 

































ANOMALY CONTRIBUTING FACTORS 


There were several factors that combined to produce the state of the spacecraft at the time 
all of the fuel was spent. This condition is referred to as the “end condition”. These 
factors were distinguished as belonging to one of two classes: factors that were necessary 
for the end condition and factors that contributed to the end condition. Those that were 
necessary are: 

1. Initial fail over, 

2. Wheel bearing friction, 

3. Safe mode transition logic, 

4. Safe mode design philosophy, 

5. Ground controller response. 

Those that were contributors are: 

6. Location of the failure in the orbit, 

7. Thruster force level. 

Each of these factors will be examined in the following section. 

Factor #1 Initial Fail Over 

The anomaly was started by what appears to be a Single Event Upset (SEU) in the on- 
board Primary Processor. The telemetry stream recorded a jump in the estimated position 
of the spacecraft at the UTC time 347/15:11:26. This position is calculated onboard to 
facilitate the nadir pointing function of the attitude control system. The change in position 
was calculated to be greater than 9888 km in 32.768 seconds. The nominal change in 
position should be around 245 km. 

After identifying and analyzing all reasonable candidates for this anomaly, it is believed 
the erroneous change in position was due to an SEU in the calculation of the spacecraft 
state (contained in the ephemeris routine). This conclusion is supported by the fact that: 

1. The magnitude of the orbit position vector is consistent between the two vectors. 
This significantly narrows the possible locations in code for the SEU to occur; and 

2. The angle between the position vectors was about 88 degrees. This error appeared 
in the pitch angle error telemetry as a value of 81.05 degrees (quaternion “small 
angle” approximation accounts for the difference). Virtually no error appeared in 
the roll or yaw angle telemetry. This suggests that the spurious position was in the 
correct orbit plane. Again, this points to a very limited number of points in the 
processing. 

Factor #2 Wheel Bearing Friction 

The initial behavior in Safe Power Mode was very nominal. This event represented the 
seventh entry into Safe Power Mode since the start of the mission and all other entries 
successfully safed the spacecraft. What made this occurrence different? The key can be 
found in the timing of the transition from the two axis controlled sun pointing inertial 
mode (Sun Point Recovery) to the spin stabilized sun pointing momentum based control 
(Long Term Hold). Initial examination of the playback data showed that there was an 


5 



anomaly in the dynamics of the spacecraft during the transition between Sun Point 
Recovery and Long Term Hold. Although there is no direct evidence of the cause because 
both the attitude decoder electronics (ADE) and the motor driver electronics (MDE) are 
turned off during Safe Power Mode, the circumstantial evidence presented below points 
to residual momentum in the wheels. 

There is a minimum delay period of ten minutes that the system must spend in Sun Point 
Recovery before it is allowed to transition to Long Term Hold. This delay was designed 
to allow for wheel rundown. Thruster activity, gyro readings and CSSA data during this 
ten minute time period give us important clues about the dynamic condition of the 
spacecraft upon attempted entry into Long Term Hold. Figure 2 shows the thruster usage 
within the ten minute delay interval. Note that only thrusters number 2 and 3 are firing 
and that they are firing in perfect unison. Thrusters 2 and 3 provide positive pitch torque 
which would be expected as the negative pitch momentum bias is transferred from the 
wheels to the spacecraft body. Figure 3 shows the spacecraft body rates in the pitch and 
yaw axes (no roll information is available in the backup mode). The shape of the pitch 
rate curve shows classic saw-tooth behavior associated with a thruster based controller 
with a fixed minimum pulse width subject to a near constant disturbance torque (due to 
the wheel run-down). 

The total angular impulse provided to the system in this ten minutes adds up to between 
2.0 and 2.25 N-m-sec. This is based on the expected force level of about 0.35 lbf per 
thruster and the telemetry data which showed 586 counts (2.93 sec) of pitch thruster 
firing. Since the wheels started with 3.0 N-m-sec of momentum at their nominal 2000 
rpm, there was 0.75 to 1.0 N-m-sec of residual momentum in the system when the 
spacecraft attempted to spin up about the roll axis. This residual momentum would 
certainly cause the “wobble” observed as the spacecraft began to spin up in roll. This is 
an unusual case where lower than expected wheel bearing drag caused the problem. 

Figure 4 is generated from on orbit data and shows a plot of the average voltage needed to 
keep the TOMS-EP wheels at 2000 rpm over the life of the spacecraft. Based on a linear 
estimate of the voltage to torque ratio, the drag seems to have leveled off at around 2 mN- 
m. Figure 5 show the results of a type A scan wheel life test performed at Ithaco over the 
course of three years. This test was performed under flight like conditions (in vacuum). 
The data shows that the drag varied from 4.25 mN-m at near beginning of life to around 
3.25 mN-m at the end of three years. The lower limit was actually established 16 months 
into the test. 

The shapes in Figures 4 and 5 are very similar. The data suggest that the wheels have 
reached a steady state and there is no reason for concern over the health of the wheels. 
The difference is the magnitude of the drop in drag torque. The test wheel showed less 
then a 25% drop in drag over a 3 year interval. The on orbit wheels show greater than a 
60% drop in torque in less than 2 years. The analysis below will show how the 
unexpectedly low drag torque caused the system to fail. 
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Thruster Activity During 10 Minute Wait 
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Figure 2 Thruster Firing Figure 3 Spacecraft Body Rates 


Estimated Drag Torque 



Figure 4 Lifetime Drag Torque (Estimated From Voltage) 
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Figure 5 Ithaco SCANWHEEL Drag Torque Life Test Data 


7 




Anomaly Simulations 

Simulations were run in an attempt to match the behavior of the anomaly. The attitude 
control and determination subsystem verification simulation (TOMSIM) was used to try 
and duplicate the behavior of the spacecraft at the time of the failure. Using initial 
conditions similar to the state of the spacecraft at the time of the failure, the transition 
from Normal Science Mode to Safe Power Mode was repeated for different levels of 
wheel bearing drag. The drag value was decreased until the system failed the transition 
from Sun Point Recovery to Long Term Hold. For reference, the top line in Figure 6 
shows the drag torque requirement imposed on Ithaco during the procurement of the 
wheels. 



Figure 6 Drag Data Used in Simulations is Derived From Max Drag Requirement 
Wheel Drag at 50% of the Maximum Allowed 

This simulation shows the expected end of life performance of the Normal Science Mode 
to Safe Power Mode transition. In this case, the wheel model used the 50% line from 
Figure 6. Figures 7-10 show the behavior of a system that has the same initial conditions 
as the anomaly. Figure 7 shows the wheel speeds. The wheel that starts near -2000 rpm is 
the +Y wheel and the wheel that starts near +2000 rpm is the -Y wheel. At 50% of the 
maximum specified friction, the wheels are run down before the 10 minute waiting period 
is finished. Figure 8 shows the spacecraft body rates. A saw-tooth pattern that is similar to 
the actual anomaly data can be seen. There is a small rate transient when the spacecraft is 
spun up in roll at around 775 seconds. Figure 9 shows the processed CSSA data which 
gives sun angles for pitch and yaw. At the time of spin-up, the pitch and yaw error do not 
exceed 5 degrees. Figure 10 shows the thruster command “on” flags. There is near 
continuous thruster activity during the spin-up but after the spin-up is completed, thruster 
usage drops to zero. 
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Wheel Drag at 20% of Maximum Allowed 

The second simulation case presented here shows what happens when there is too much 
momentum in the system at the time of roll spin-up. Figures 1 1-14 show the behavior of a 
system that has the same initial conditions as the anomaly but wheel drag is scaled to 
20% of maximum. Figure 1 1 shows the wheel speeds. At 20% drag, there is still 800 rpm 
(1.2 N-m-sec) remaining in the wheels when the spacecraft begins to spin up about roll. 
Figure 12 shows the spacecraft body rates. Coning and nutation are now apparent in the 
motion of the spacecraft. Figure 13 shows the processed CSSA data which gives sun 
angles for pitch and yaw. The system is unable to complete the transition from Sun Point 
Recovery to Long Term Hold because the processed sun angle error is too large. Figure 
14 shows the thruster command “on” flags. Since the spacecraft was unable to complete 
the transition to the momentum based controller, the system is now using a two axis 
inertial sun pointing control law (Sun Point Recovery) with a high roll rate. This 
controller is unsuited for systems with a large momentum bias and the pitch thrusters 
begin to fire continuously in a futile attempt to reduce the observed pitch rate (caused by 
misalignment of control and inertial axes and the presence of a significant roll rate). The 
combination of very small misalignments in the thrusters and CG migration over the life 
of the spacecraft caused a slight pitch/roll torque coupling. As the pitch thrusters 
continued to fire, the roll rate slowly increased to 18 deg/sec at which point the 25 lb of 
hydrazine was exhausted. 


Reconstruction With '0% o( Mcnnvim Specific*! Wived FriclHHk 


ReLOnrtnKUon With 20% of Mumum Specific*! Wheel Friction 




Figure 11 Wheel Spin Down Figure 12 Spacecraft Body Rates 
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Figure 13 Processed Sun Sensor Angle Figure 14 Thruster Commands 
Factor #3 Safe Mode Transition Logic 

The National Transportation Safety Board (NTSB) approaches investigations with the 
motto “If any link in the chain of events were broken, the accident would not occur.” Of 
all the contributors, the safe mode transition logic would have been the easiest link to 
break. The crux of the problem is this: a control flag was used for two purposes, both to 
turn on and off the momentum controller and to signal the end of the roll spin-up 
maneuver. The Sun Point Recovery logic interfered with the function of the roll spin-up 
logic, thus preventing the transition to the momentum control mode. 

There are two flags of interest that control the transition to Long Term Hold. These two 
flags are named “runup” and “isunon” (integer sun control on/off flag). In a nominal 
scenario, the flight software should go through the procedure outlined below: 

1. Sun Point Recovery is entered, “runup” is initialized to 0 and “isunon” is 
initialized to 1. 

2. The spacecraft tries to satisfy the five conditions listed in Table 2 by acquiring the 
sun and becoming quiescent. 

3. When the five conditions are satisfied, “runup” is set to 1. 

4. If ten or more minutes have passed, the necessary roll thrust is calculated and the 
roll thrusters begin to spin up the spacecraft, “runup” is set to 2. 

5. When the spin-up is complete, “isunon” is set to 0 to tell the mode transition logic 
to transition to Long Term Hold. 

6. Long Term Hold is initialized with the controller off (“isunon” = 0) and it is 
usually a day or two before the controller needs to be turned on. 
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The potential flaw in this logic comes from the fact that the mode transition logic runs at 
1.024 second intervals and the Sun Point Recovery controller runs at 0.256 second 
intervals. There is potential delay between when the “isunon” flag is set to 0 and the 
mode transition logic reads it. That delay may be anywhere from 30 msec to 798 msec. In 
that time, the Sun Point Recovery controller may be run either 0,1,2 or 3 times. If the sun 
angle is outside the 12 degree outer deadzone, the control logic will set the “isunon” flag 
back to 1 before the mode transition logic can read it. 

This logic was tested extensively in both simulation and fixed based test without 
discovering this potential flaw. That is because the spin-up process does not start unless 
the angle error is within the 5 degree deadzone and the spacecraft is under active position 
and rate control during the spin-up. The transition was simulated using worst case thruster 
misalignments, force mismatch, CG offsets, force vector rotations and nozzle exit 
location errors. In all cases, the transition to Long Term Hold was achieved. 

Factor #4 Safe Mode Design Philosophy 

In order to insure that the cause of an anomaly is removed from the system and to 
eliminate software health checks for equipment, it was decided to use all standby 
redundant components for the thruster based safe modes. Because of budget constraints, it 
was impossible to meet the criteria for using all standby redundant equipment in Safe 
Power and have rate information for all three axes. The choice was made to use a single 
gyro in the backup mode and maximize the control stability in other ways. 

As many “smart” decisions as possible were made to mitigate the lack of roll rate 
information: 

1. Point major moment of inertia at Sun. 

2. Use a two-stage safing procedure. The first mode (Sun Point Recovery) is 
temporary and once the Sun is acquired, the system transitions to a spin stabilized 
mode that used a momentum controller. 

3. The momentum controller was designed to be stable over a wide range of roll 
rates. Simulation has shown this controller to be stable from 0.75 to > 20 deg/sec. 

4. Bias rejection filters were added to remove DC signals associated with roll rate. 

5. Control only executed at orbit location where “Earth shine” is minimum. 

6. A failsafe check will return to two axis inertial control if momentum controller 
failed to hold Sun. 

It was known that the two axis inertial mode would use a large amount of fuel if 
significant roll rate accumulated. This was an acceptable risk to the program since the 
system was designed to pass through this mode in a short period of time. 

Factor #5 Ground Controller Response 

Once the anomaly occurred, TOMS-EP consumed fuel at a very high rate for a period of 
approximately 6 hours before the fuel was depleted. Ground controllers had only 4 
contacts during this time, the first of which was only 3 minutes long by the time the 
ground acquired a signal. The other 3 passes were on the order of 10 minutes each. 
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During the 4 passes, the ground had the ability to disable the thrusters, which in hind 
sight would have saved fuel and prevented such a high spin-up. 

Although ground controllers could have prevented or minimized the effects of the 
anomaly, it is understandable why they were unable to do so. Although tank pressure was 
dropping and the spacecraft was spinning up about roll, TOMS-EP remained in the proper 
Sun-pointing attitude at each contact. Furthermore, while in Safe Power Mode, there is 
not a direct measurement of roll rate available in telemetry. Further complicating 
understanding at the time was the fact that when ground controllers turned on other gyros 
to look at the roll rate, the spacecraft processor reset during the same pass. It was later 
determined that this was just a coincidental 7000 second timer reset which had nothing to 
do with turning on gyros. 

Factor #6 Location of Failure in the Orbit 

The location of the failure had a significant role in the behavior of the system for two 
reasons. First, after the Redundant Processor had booted up, the spacecraft was almost 
exactly sun pointed. Figure 15 shows the processed CSSA angles over the entire 10 
minute wait period. The maximum angle observed was just above 3 degrees. At the end 
of the wait period, the spacecraft immediately began to spin up. Second, the presence of 
Earth shine fooled the spacecraft into thinking it was still sun pointed even after rotating 
more than 20 degrees. If the spacecraft pitch rate shown in Figure 8 is integrated, it 
should produce a change in pitch angle as shown in Figure 16. Figure 17 shows the 
processed CSSA angles with the effects of Earth shine removed. These data match the 
integrated gyro data much more closely. It is quite possible that in the absence of Earth 
shine, the system would not have satisfied the five conditions for spin-up immediately 
and the reaction wheels would have had more time to spin down. 


Processed CSSA During 10 Minute Wait 
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Figure 15 Processed CSSA Sun Angles 
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Figure 16 Integrated Rates Figure 17 Improved Estimate of CSSA Angle 


Factor #7 Thruster Force Level 

The last contributor identified was the level of force available at the time of the anomaly. 
Due to of the initial orbit insertion bums and subsequent orbit change, over 90% of the 
hydrazine fuel had been exhausted from the blow down propulsion system and the force 
level was around 35% of the force level available at the beginning of life. The lower force 
level made the system less capable of countering the (OxH torques generated by the 
residual wheel momentum. 


RECOVERY EFFORT 

The focus of the recovery effort was to generate a scenario that was easily implemented 
and that maximized the probability of spacecraft recovery. To be successful, the recovery 
must first and foremost maintain the health of the power subsystem. 

Power Subsystem Considerations 

The original design of the power subsystem makes it very robust to attitude anomalies. 
The fixed arrays are arranged in a cruciform orientation off of the -Z body axis as shown 
in Figure 1. Figure 18 shows the power output of the arrays as a function of the solar 
normal vector (neglecting shadowing effects) scaled to the output of a single sun pointed 
array. The power system produces enough power to run the spacecraft in all orientations 
except when the sun is within about 45 degrees of the plus or minus Y spacecraft axis. 
Since the Y axis is the intermediate axis of inertia, the Sun should not dwell near the axis 
if there is any significant angular rate in the spacecraft body. 

The TOMS-EP battery has 9 amp-hours of capacity. With normal loads, the battery can 
sustain the spacecraft for about 3 hours without solar array power. It should be noted that 
there is no provision for “jump starting” the power system after the battery is discharged 
since the solar array regulators (SARs) are powered from the battery. 
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Normalized Solar Array Power Potential 



Figure 18 Normalized Power Output For Different Sun Vector Orientations 


Attitude Control and Determination System Considerations 

The Attitude Control and Determination System (ACDS) was essentially disabled after 
the fuel was exhausted. A task was immediately undertaken to ascertain the current state 
of the spacecraft and predict the future orientation with respect to the sun. 

The gyros could not be used for on-board roll rate determination because the spacecraft 
was spinning at 18 deg/sec and the gyros lose polarity at 7 deg/sec. At the beginning of 
the anomaly, the Sun was within the field of view of one of the FSS; the other FSS was 
pointed anti-Sun. From the CSSA, it was known that the spacecraft +X axis was pointed 
within about 5 degrees of the Sun and moving away at a rate of approximately 2-3 
degrees per day. Although data from the magnetometer was available, the absolute 
inertial attitude was very difficult to determine due to the interaction of the high roll rate 
and processing and telemetry delays. It was certain that if the precession rate observed on 
the CSSAs continued, the power system would see significant reductions in available 
power within 2 weeks. 

With so little time available, the recovery procedure had to be designed and tested within 
10 days. This requirement drove the team toward trying to use the onboard algorithms 
with minimum modifications. Looking at a block diagram of the TOMS-EP ACDS 
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hardware shown in Figure 19, it can be seen that wheels and torque rods are the only 
actuators available for maneuvering the spacecraft once fuel is depleted. It was not 
possible to use the wheels for a large angle maneuver since each was only capable of 
containing 4 N-m-sec of momentum and the spacecraft body contained about 36 N-m-sec 
of momentum. The torque rods could be used to slowly maneuver the spacecraft if there 
was sufficient time. Fortunately, there are two onboard magnetic control algorithms 
onboard to choose from. The first is a cross product law used for momentum unloading 
and the second is a B-dot law used for a magnetic Safe Hold Mode. This was the extent 
of the tools to be used in the recovery. 



Figure 19 ACDS Equipment Block Diagram 
Thermal Subsystem Considerations 

The thermal subsystem was designed to radiate most of the heat generated in the 
spacecraft out the panels on the -Y spacecraft axis. The constraint this placed on the 
recovery scenario was essentially enveloped by the power system requirements. 

Propulsion Subsystem Considerations 

Although it was thought that all the fuel in the tank was exhausted, there was known to be 
fuel in the prime side thruster lines. In addition, there may have been some fuel trapped 
by the tank bladder against the side of the tank. Immediately after the thruster valves on 
the redundant side were closed, the pressure in the tank began to rise slowly. Although 
the recovery could not count on using the impulse from the trapped fuel, an attempt could 
be made to use it in the most constructive manner possible. When the spacecraft was 
returned to the prime processor, the Long Term Hold momentum control was selected 
and the residual propellant precessed the spin vector nearly 10 degrees towards the Sun. 
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This “last gasp” contribution from the propulsion subsystem gave the recovery team 
several more days to plan the recovery procedure. It also allowed the team to turn off all 
propulsion heaters and save power at critical points in the recovery. 

RECOVERY PLAN 

Of the two onboard magnetic control laws, the B-dot law would need the least amount of 
modification because it is primarily a minimum energy based design. In the absence of 
internal momentum, a spacecraft in a polar orbit using a B-dot control law will eventually 
end up with the maximum moment of inertia (roll) perpendicular to the orbit normal and 
the roll body rate would approach 2 revolutions per orbit (RPO). If the wheels were spun 
up to produce momentum in the -Y spacecraft axis, this momentum would end up 
perpendicular to the orbit plane with 2 RPO rate about the pitch axis. 

In this instance, knowing the start and end conditions did not answer the question of 
whether the spacecraft would pass through an unfavorable power condition somewhere in 
between. If the wheels were running, the Y axis could act as a pseudo maximum moment 
of inertia and it is possible for the Sun vector to remain near the Y axis long enough to 
discharge the battery. The best way to minimize this possibility was to break the recovery 
into three stages: 

1 . B-Dot magnetic despin without internal momentum, 

2. Wheel capture into the nominal Safe Hold Mode, 

3. Science Return into nadir pointing. 

Simulations 

The entire recovery scenario was extensively simulated using TOMSIM prior to the start 
of the spacecraft recovery attempt. These simulations calculated both attitude and power 
potential. Direct measurements from the CSSA provided data on the angle between the 
spacecraft X axis and the Sun vector. That narrowed the uncertainty in the spin axis 
attitude to the surface of a cone about the Sun. After the recovery scenario was 
established, the robustness of the recovery approach was examined by simulating the 
process using four different sets of initial conditions that resided on the surface of the 
uncertainty cone. The simulated recovery was successful in all four cases. 

Figure 20 shows the spacecraft body rates for a simulated recovery from a position 30 
degrees east of the Sun. The simulation predicted a stage 1 duration of approximately 2 
days. Although the roll rate could not be directly measure during the actual recovery, 
Figure 21 shows the rate estimated from the DC bias on the pitch gyro which should be 
proportional to the roll rate if the system is in a flat spin and has roll-pitch cross products 
of inertia. 
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Figure 20 Simulated Recovery Rates Figure 21 Estimated Recovery Rates 
Stage 1 

The goal of the first stage was to reduce the spin rate from 18 deg/sec to within the 
capture range of the Safe Hold Mode (between 2-3 deg/sec). The normal B-dot algorithm 
processing is executed every 16.384 seconds which allows multiple magnetometer 
samples to be averaged for noise reduction. In addition, the magnetic field rate is 
calculated with a differential filter that has a time constant longer than 16.384 seconds. 
Clearly, the algorithm could not be used successfully with the spacecraft spinning at 18 
deg/sec. Changes in the database allowed us to successfully reduce the number of 
magnetometer samples to 1 and change the characteristics of the differential filter. The 
next step involved changing the flight software executive to call the B-dot algorithm 
every 2.048 seconds. Fortunately, this was accomplished by replacing a single byte in an 
inequality statement. The final “high speed” B-dot algorithm had 1.024 seconds allocated 
to magnetometer sampling and 1.024 seconds allocated to torque rod firing. 

The question still remained whether the magnetic despin would put the spacecraft in an 
unfavorable power attitude. If you base your guess on the known end condition, you 
might assume that the maximum moment of inertia would be pushed perpendicular to the 
orbit plane. In fact, just the opposite is true. Since the body rates are much higher than 
orbit rate, the B-dot algorithm simply sheds energy wherever it can. The key to 
understanding its behavior is in the available torque. In a near polar orbit, the magnetic 
field remains close to the orbit plane (rotating twice per orbit). If you break the total 
system momentum into the portion projected into the orbit plane and the portion 
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perpendicular to the orbit plane, there is always magnetic torque available to reduce the 
momentum perpendicular to the orbit plane and four points in the orbit where it is very 
difficult to affect the momentum in the orbit plane. Thus as long as the body rates remain 
high relative to orbit rate, the maximum moment of inertia will remain close to the orbit 
plane. 

Since TOMS-EP resided in a sun synchronous orbit with an 11:00 to 11:30 ascending 
node, pulling the maximum moment of inertia toward the orbit plane should not degrade 
the power potential. 

Stage 2 

Stage 2 was the riskiest portion of the recovery scenario. At some point, the spacecraft 
had to transition from spinning about the maximum moment of inertia to spinning about 
the Y axis. The transition had the potential of pointing the spacecraft in a low power 
attitude for an extended period of time. 

Once TOMS-EP was despun to 3 deg/sec, the nominal B-Dot Mode software parameters 
were restored. The wheels were set to their minimum rotation rate to minimize the time in 
transition between sun pointing and normal B-dot pointing. The spacecraft was prepared 
for the low power attitude by shedding all loads not directly involved in the recovery. 
These loads included prime and redundant platform heaters, prime and redundant 
propulsion heaters, gyros, and all transmitters. In this configuration, the average load was 
reduced to 45 watts (1/3 of orbit average). Essentially, it was up to the physics of the B- 
dot controller to complete the transition. Interference from the ground would only reduce 
the chance of a successful recovery. 

Two orbits after starting stage 2, contact was re-established with the spacecraft. The 
system had settled with the pitch momentum bias perpendicular to the orbit plane. The 
wheel speeds were slowly increased until they matched the normal Science Mode speeds. 

Stage 3 

In the final Science return stage, TOMS-EP was commanded into its originally designed 
Science Return Mode. Although never used prior on-orbit, the mode was well tested and 
simulated prior to delivery of TOMS-EP. This mode allowed automatic transition into 
Science Mode within one orbit. 

POST RECOVERY FAULT MANAGEMENT IMPLICATIONS 

The loss of the propulsion subsystem left the Safe Power Mode incapable of active 
control. If the spacecraft switches to an uncontrolled mode, it is known that the 
momentum stored in the pitch momentum bias will eventually end up as a roll rate of +/- 
1.9 deg/sec. This should be sufficient to prevent complete battery discharge. 

It is still preferred to keep under active control if possible. For this reason, the onboard 
fault detection software was modified to minimize the number of faults that send the 
system to Safe Power Mode. Only those faults that require reconfiguration of power 
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system or processor faults send the system to Safe Power Mode. All other faults (pointing 
anomalies, wheel speed delta, etc.) cause a transition to the B-dot Safe Hold without 
switching processors. 

LESSONS LEARNED 

1. SEUs happen. Be ready. 

2. Where it is possible, directly measure states that are driving decisions. In our case, it 
would have been preferable to measure the wheel speeds directly and start sun acquisition 
after they had completed their run down. Unfortunately, that was not possible because 
tachometer data is unavailable when the motor drivers or the attitude decoder electronics 
are off. Additional fault management risk would have been assumed if the wheel 
electronics were left on in Safe Power Mode. 

3. Designers often concentrate on accommodating behavior associated with “worst case” 
conditions (CG locations, misalignments, friction, structural flexibility etc.). Sometimes, 
an ideal CG location, perfect alignments, better than expected friction or higher than 
expected stiffness can cause problems. These should be considered also. 

4. Do not use flags for multiple purposes no matter how closely related they are. 
Carefully check the logic of flags that are set and read asynchronously. 

5. The emphasis in fault management at TRW has shifted since the design of TOMS-EP. 
In subsequent programs, the inherent robustness of the safe mode was considered to be 
more important than using standby redundant components. Even in light of the TOMS-EP 
on orbit experience, this is not a clear cut decision. To address the issue of using a single 
gyro for safe modes on the new programs, flight software chooses which pair of gyros to 
turn on based on a number of comparison tests. It has been demonstrated in fixed based 
test that these tests can be fooled under certain circumstances. If the software chooses a 
failed gyro, the consequences will be worse than the TOMS-EP anomaly. 

6. Murphy works smarter than you do. Rely more on general principles to prove system 
robustness rather than attempting to find degenerative cases. 

7. A number of spacecraft have been lost or nearly lost due to anomalous autonomous 
thruster operation. If a spacecraft has the capability to use wheels rather than thrusters to 
acquire its safe mode orientation, then it is usually prudent to use wheels over thrusters. 
Although thruster hardware and the associated electronics are very reliable, a system 
using expendables always introduces a risk of imparting unwanted high momentum to the 
spacecraft. This high momentum input can be caused by spacecraft hardware, logic or 
software anomalies. A wheel-based safe mode limits the amount of spin-up while in the 
mode, should an anomaly occur. 

8. The canted, double-sided solar array orientation on TOMS-EP is very forgiving of 
spacecraft attitude anomalies. Adequate power can be generated from most spinning or 
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tumbling conditions. This was of great relief while TOMS-EP was spinning uncontrolled 
for 18 days. Multiple solar array viewing angles increase the robustness of a spacecraft to 
anomalies. 

CONCLUSIONS 

The TOMS-EP spacecraft was successfully recovered in less than 3 weeks from a severe 
anomaly that depleted all fuel and left the spacecraft uncontrolled with a high spin rate. 
A team of engineers and spacecraft operators quickly determined the cause of the 
anomaly and implemented a recovery effort. The TOMS-EP satellite continues to 
successfully perform its mission well beyond its design life of mapping the global 
distribution of Earth’s atmospheric ozone layer. 


21 



